"From Elastic Net to Skynet"¶
What we will talk about¶
- Elastic net and other machine learning methods
- The philosophy beyond machine learning-based analyses
What we will NOT talk about:¶
- "Skynet" from the Terminator movies
- Advanced machine learning models
- complex deep learning architecturesd
Sorry for clickbaiting you!
AIM of this python Notebook:¶
- Establish a ground for understanding by refreshing your memories on linear regression
- Understand the difference between inference and prediction
- Demonstrate the problem of overfitting
- Understand the logic beyond cross validation
- Understand of how some classical machine learning methods can solve the issue of overfitting
- Overview of some simple machine learning techniques (neural networks, deep learning)
- Show a research example.
Python Notebook-based presentation¶
- figures are interactive
- source code is provided
- source code can be changed and run again on the fly
- feel free to interrupt the talk and patch the code if you have questions
Some technical details (not always shown in the slides).
In [1]:
# Imports and Co
import numpy as np
import pandas as pd
import plotly.io as pio
import plotly.figure_factory as ff
import plotly.offline as py #replaces import plotly.plotly as py
py.offline.init_notebook_mode()
import plotly.graph_objs as go
import plotly.express as px
from plotly.subplots import make_subplots
import teaching.teaching.plot as myplot
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import LeaveOneOut
from sklearn.preprocessing import StandardScaler
# Set default renderer
pio.renderers.default = 'notebook+jupyterlab' # See [1]
# Set default template
pio.templates['slides'] = go.layout.Template(layout=dict(width=800, height=550))
pio.templates.default = 'plotly+slides' # See [2]
In [2]:
df_full = pd.read_csv("../data/movie_metadata.csv")
df_full=df_full[['movie_title', 'title_year', 'budget', 'gross', 'imdb_score', 'duration', 'director_name', 'movie_facebook_likes',
'facenumber_in_poster', 'actor_1_facebook_likes', 'actor_1_name']]
df=df_full[:100]
df
Out[2]:
Feature transformations¶
In [3]:
df = df.dropna()
fig = make_subplots(rows=1, cols=2)
fig.add_trace(go.Histogram(x=df["budget"], name="original"), row=1, col=1)
df['gross (log)']=np.log(df['gross'])
df['budget (log)']=np.log(df['budget'])
fig.add_trace(go.Histogram(x=df["budget (log)"], name="log transformed"), row=1, col=2)
fig.update_layout(title_text='Budget $', height=300)
fig.show()
Can we predict the Gross income?¶
In [4]:
fig = px.scatter(df, x="budget", y="gross")
fig.show()
A possible "prediction": movie will have "zero balance".¶
In [5]:
fig = myplot.plot_scatter_with_regline(df, x="budget", y="gross", legend=('movie', 'zero balance'))
fig.show()
In [6]:
reg = LinearRegression().fit(X=df["budget"].values.reshape(-1,1), y=df["gross"])
print("Intercept:", reg.intercept_, "Slope:", reg.coef_)
print("Predicted income of a 1.000.000$ movie:", reg.predict(np.array([[1000000]])) )
In [7]:
predicted = reg.intercept_ + reg.coef_*df["budget"]
#predicted = reg.predict(df["budget (log)"].values.reshape(-1,1))
fig = myplot.plot_scatter_with_regline(df, x="budget", y="gross",
predicted=predicted, legend=('movie', 'linear fit'))
fig.show()
Multiple explanatory variables (features)¶
In [8]:
df
Out[8]:
Multiple Linear Regression¶
In [9]:
reg_multiple = LinearRegression().fit(X=df[['budget', 'duration', 'title_year', 'movie_facebook_likes']], y=df["gross"])
print("Intercept:", reg_multiple.intercept_)
print(' budget (log)', 'duration', ' title_year', ' movie_facebook_likes')
print("Slope:", reg_multiple.coef_)
print("Predicted income of a movie with:\n","budget=1.000.000$\n","duration: 120\n","year: 1996\n","likes: 100000\n",
"=", reg_multiple.predict(np.array([[1000000, 120, 1996, 100000]])), "$" )
In [10]:
predicted_multiple = reg_multiple.predict(df[['budget', 'duration', 'title_year', 'movie_facebook_likes']])
fig = myplot.plot_scatter_with_regpoints(df, x="budget", y="gross",
predicted=predicted_multiple, legend=('movie', 'linear fit'))
fig.show()
Predicted vs. Observed plots, prediction error¶
In [12]:
fig.show()
In [13]:
print("MSE:", mean_squared_error(df["gross"], predicted), mean_squared_error(df["gross"], predicted_multiple))
Prediction error improved as we added more features¶
- why not adding many features?
- Lets do an experiment: add many random features
In [14]:
def simulate(n_random):
return np.hstack((df['budget'].values.reshape(95,1), np.random.normal(0,1,(len(df['budget'].values), int(n_random)))))
mse=[]
for i, num_rand in enumerate(np.linspace(0, 79, 80)):
X=simulate(num_rand)
reg_random = LinearRegression().fit(X=X, y=df["gross"])
mse.append(mean_squared_error(df["gross"], reg_random.predict(X)))
if i%10 == 0: print(mse[i])
In [15]:
fig = px.scatter(x=np.linspace(0, 79, 80), y=mse, trendline="ols", labels={'x':'num random variables', 'y':'MSE'})
fig.show()
What on earth is happening?¶
- More explanatory variables -> more beta values to set
- A model with a sufficient number of (beta) parameters is able to "learn" the data at hand, instead of the relationship
BUT:
- Such a model will not generalise to new data
- Predictive power will be low
This is called:
Overfitting¶
Another example: adding ith degree of the variable¶
Real examples of overfitting¶
Google Flu Trends in 2009
Real examples of overfitting¶
RMS of models random variables on hold out data¶
In [16]:
df_train = df
df_test = df_full[100:200]
df_test = df_test.dropna()
def simulate(df, n_random):
return np.hstack((df['budget'].values.reshape(len(df),1), np.random.normal(0,1,(len(df['budget'].values), int(n_random)))))
mse_train, mse_test = [], []
for i, num_rand in enumerate(np.linspace(0, 79, 80)):
X_train=simulate(df_train, num_rand)
reg_random = LinearRegression().fit(X=X_train, y=df["gross"])
X_test=simulate(df_test, num_rand)
mse_test.append(mean_squared_error(df_test["gross"], reg_random.predict(X_test)))
if i%10 == 0: print(mse_test[i])
In [17]:
fig = make_subplots(rows=1, cols=2)
fig.add_trace(go.Scatter(x=np.linspace(0, 79, 80), y=mse, mode='markers', name="MSE on train"), row=1, col=1)
fig.add_trace(go.Scatter(x=np.linspace(0, 79, 80), y=mse_test, mode='markers', name="MSE on test"), row=1, col=2)
fig.update_layout(title_text='Adding random variables', height=300)
fig.show()
How can we know what is the optimal lambda?¶
In [18]:
loo=LeaveOneOut()
for train, test in loo.split(df):
print('train:', train)
print('test: ', test)
Ridge¶
In [20]:
X = simulate(df, 10)
y = StandardScaler().fit_transform(df["gross"].values.reshape(-1,1))
alphas = [1, 5, 50, 100]
mse=np.zeros((loo.get_n_splits(df), len(alphas)))
idx=0
for train, test in loo.split(df):
for i, alpha in enumerate(alphas):
ridge = Ridge(alpha=alpha).fit(X=X[train], y=y[train])
mse[idx, i] = mean_squared_error(ridge.predict(X[test]), y[test])
idx += 1
pd.DataFrame(mse, columns=alphas).mean()
Out[20]:
In [21]:
print(Ridge(alpha=1).fit(X=X, y=y).coef_.round(3).tolist())
print(Ridge(alpha=5).fit(X=X, y=y).coef_.round(3).tolist())
print(Ridge(alpha=50).fit(X=X, y=y).coef_.round(3).tolist())
print(Ridge(alpha=100).fit(X=X, y=y).coef_.round(3).tolist())
LASSO¶
In [25]:
np.random.seed(1)
X = simulate(df, 10)
y = StandardScaler().fit_transform(df["gross"].values.reshape(-1,1))
alphas = [0.0001, 0.001, 0.01, 0.1, 0.2]
mse=np.zeros((loo.get_n_splits(df), len(alphas)))
idx=0
for train, test in loo.split(df):
for i, alpha in enumerate(alphas):
lasso = Lasso(alpha=alpha).fit(X=X[train], y=y[train])
mse[idx, i] = mean_squared_error(lasso.predict(X[test]), y[test])
idx += 1
pd.DataFrame(mse, columns=alphas).mean()
Out[25]:
In [26]:
np.random.seed(1)
print(Lasso(alpha=0.0001).fit(X=X, y=y).coef_.tolist())
print(Lasso(alpha=0.001).fit(X=X, y=y).coef_.tolist())
print(Lasso(alpha=0.01).fit(X=X, y=y).coef_.tolist())
print(Lasso(alpha=0.1).fit(X=X, y=y).coef_.tolist())
print(Lasso(alpha=0.2).fit(X=X, y=y).coef_.tolist())
In [28]:
import tensorflow as tf
from tensorflow import keras
def preprocess(x, y):
x = tf.cast(x, tf.float32) / 255.0
y = tf.cast(y, tf.int64)
return x, y
def create_dataset(xs, ys, n_classes=10):
ys = tf.one_hot(ys, depth=n_classes)
return tf.data.Dataset.from_tensor_slices((xs, ys)) \
.map(preprocess) \
.shuffle(len(ys)) \
.batch(128)
# get data
(x_train, y_train), (x_val, y_val) = keras.datasets.fashion_mnist.load_data()
train_dataset = create_dataset(x_train, y_train)
val_dataset = create_dataset(x_val, y_val)
In [34]:
# set up
model = keras.Sequential([
keras.layers.Reshape(target_shape=(28 * 28,), input_shape=(28, 28)),
keras.layers.Dense(units=256, activation='relu'),
keras.layers.Dense(units=192, activation='relu'),
keras.layers.Dense(units=128, activation='relu'),
keras.layers.Dense(units=10, activation='softmax')
])
In [35]:
model.compile(optimizer='adam',
loss=tf.losses.CategoricalCrossentropy(from_logits=True),
metrics=['accuracy'])
history = model.fit(
train_dataset.repeat(),
epochs=5,
steps_per_epoch=200,
validation_data=val_dataset.repeat(),
validation_steps=2
)
